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MOTIVATION 


•  Joint  Capabilities: 

-  Future  operating  environment: 
uncertainty,  complexity,  rapid 
change,  and  persistent  conflict. 

-  Requires  integrated  approach 
(WSARA  2009) 

•  Growing  PE  /  MDAP 
interdependencies  and 
complexity. 

-  Focus  on  funding  interdependency 
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RESEARCH  PATHS 


•  Network-centric  approach  (Brown  and  Owen,  2012;  Raja  et  al., 
2012) 

•  Automated  analysis  for  correlation  and  associations 

(Zhao  et  al,  2012) 

•  Portfolio-based  approach  (Davendralingam  et  al.) 
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OUR  PREVIOUS  WORK  (rajaetal., 2012, 2013): 

•  Data-driven  approach  to  develop  what-if  models  to  predict  early  indicators  of 
cascading  risks. 

•  DAES  reports  of  small  set  of  MDAPs  over  several  years. 

•  Results: 

-  Non-local  factors  affect  program  outcomes:  “program-centric”  +  “program 
network  approach”  for  acquisition  and  management. 

-  Cascading  effects  recast  as  a  sequential  decision  problem. 

-  Tedious  manual  process  to  build  decision  process  model. 


MDAP  A 


CASCADING  EFFECT:  HIDDEN 
DYNAMICS 
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CURRENT  RESEARCH  GOAL 


•  Automate  Extraction  &  Analysis  of  MDAP  DAES  PSM, 
Issues,  Actions: 

-  DAES  data  for  multiple  programs  over  multiple  years. 

•  Automate  Extraction  of  Structural  Properties  of  MDAP 
Network: 

-  Identify  funding  neighbors  based  on  PE  and  SAR  data. 

-  Determine  link  weights  from  PE  perspective. 

•  Populate  Decision  Process  Model. 

•  Identify  challenges  to  data  acquisition.  ♦ 
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DECISION  SUPPORT  FRAMEWORK 
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MAIN  CONTRIBUTIONS 


•  Network  and  Program  Centric  analysis. 

•  Novel  Integration  of  methodologies  for  text  and  image  analytics 
for  large  scale  automated  data  extraction. 

•  Quantification  of  interdependency  metric. 

•  Progress  towards  Decision  support  framework  for  MDAP 
cascading  risk  prediction. 
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INPUT  DATA :  PROGRAM  STATUS 
MATRIX 
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INPUT  DATA:  ISSUE  SUMMARY  -  9 

4  SAMPLE  RECORDS 


ISSUE  SUMMARY 

ISSUES 

ACTIONS 

Schedule-The  reflectors  are  undergoing 
design  modifications  as  a  result  of 
thermal  predicts  that  exceed  allowable 
limits  and  PIM  out  of  spec  performance 

Program  Office  is  working  to  amend  the 
contract  and  the  incentive  structure  to 
maximize  the  likelihood  for  achieving  the 
contractor's  commit  date 

Cost  Control-  A  Team,  has  been  unable  to 
provide  an  accurate  forecast  of  projected 
cost. 

XYZ  has  detailed  manpower  projections 
and  is  measuring  each  organization's 
effectiveness  in  reducing  headcount,  a 
metric  considered  key  to  managing  costs 

lARLOITE 
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MDP  STATE  FEATURE  MATRIX 
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Feature  Number 

Feature  Description 

An  example  of  Feature  value 

Feature  1 

Program  ID 

M  DAP_A 

Feature  2 

Current  Year 

2010 

Feature  3 

Current  Month 

April 

Feature  4 

Cost(APB)  status,  for  9  months  starting  with 
current  month 

<1,1,1, 1, 1,1,1, 1,1,1,1,1> 

Feature  5 

Cost(Contract)  status,  for  9  months  starting 
with  current  month 

<0,0,0,-l,-l,-l,-l,-l,l,l,l,l> 

Feature  6 

Schedule(APB)  status,  for  9  months  starting 
with  current  month 

<0,0,0, 0, 0,0,0, 0,0,0,0,0> 

Feature  7 

Schedule(Contract)status,  for  9  months 
starting  with  current  month 

<0,0,0,-l,-l,-l,-l,-l,l,l,l,l> 

Feature  8 

Performance  (APB)  status,  for  9  months 
starting  with  current  month 

<1,1,1, 1, 1,1,1, 1,1,1,1,1> 

Feature  9 

Performance(Contract)  status,  for  9  months 
starting  with  current  month 

<1,1,1, 1, 1,1,1, 1,1,1,1,1> 

Feature  10 

Funding(APB)  status,  for  9  months  starting 
with  current  month 

<1,1,1, 1, 1,1,1, 1,1,1,1,1> 

w 

Feature  11 

Funding(Contract)  status,  for  9  months 
starting  with  current  month 

<1,1,1, 1, 1,1,1, 1,1,1,1,1> 

UUUITE 

Feature  12 

List  of  Issues 

Cost  unable  to  forecast 

Feature  13 

List  of  Actions 

Reduce  resources 

1  CLAIM 
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Issue  Summary  (Chart  2) 


Issue/Problm  Description 

Action  Plan 
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LATENT  DIRICHLET  ALLOCATION(LDA 

•  A  Document  is  a  bag  of  words;  random  mixture  of  latent  topics 

•  Unsupervised  learning  -  uncover  the  latent  topics  characterized  by  a 
statistical  distribution  in  a  given  set  of  document. 

•  Result  in  distribution  of  1)  topics  across  documents  and  2)  words  across 
topics. 

•  Training  data:  DAES  reports  of  MDAP_Afrom  2007-2011  with  150  records 


Farmers  who  received  some  government  information  late 
ha ve  bee n  given  additional  to  fi I e  the i r  1987  federa I 
return.  The  internal  service  said 

it  was  special  relief  to  Farmer's  who  did  not  receive 

agriculture  department  documents  by  the  Feb  15  scheduled 
and  Form  1099-A,  on  which  s  or 

abandcr  mer  of  secured  =-  are  reported. 


Topic  0  Topic  1  Topic  2 


Topic  3 


David  M.  Blei, 
of  Machine  Learni 


and  Michael  I.  Jordan,  Latent  Dirichlet  Allocation, 
“,  Pages  993-102?!  March  2003 
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MODEL  PARAMETER  ESTIMATION 


Perplexity  is  a  measure  of  model’s  ability  to  infer  the  topics  in  unseen 
documents. 


K=  15  for  issues;  10  for  actions. 


Estimation  of  k  for  ISSUES 


“Run  1 
"Run  2 
“HU  fli 
"Hun  4 
“Run  5 
■  Average 


5  10  IS  20  25  30  55  40  45  SC-  55  60  65  70 

k-MumbcrcfTopici 


Estimation  of  k  for  ACTIONS 


a  10  20  30  40  50 

k-  Number  of  Topics 
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ISSUE  AND  ACTION  TYPES 


14 


Logistics 


e  Description 


Repeated  inability  to  accurately  forecast 
cost 

Overrun  costs  due  to  technical  delays 

Modifications  to  total  cost  to  avoid  going 
over  budget 

Overrun  costs  due  to  unanticipated  extra 
elements  in  the  project 
Hardware  Related  Issues 


Type  2  Software  Related  Issues 

Type  3  International  Collaboration  Delay 

Type  4  Execution  Delay  due  to  failure  to  meet 
standards 

Type  5  Funding  Delay 

Type  1  Ground  Site  is  unavailable  for  installation. 

Type  1  Funding  has  been  aligned  with  estimate 

Type  2  Funding  is  short 

Type  3  Funding  has  been  frozen 

Type  4  Funding  is  being  reassessed  due  to  errors 
or  changes  in  need 


ACTION 


Schedul 


Funding 


Type  5 


Description 


Contractor  Renegotiation 

Engage  with  contractor  to  address  cost 
issues 

Monitor  performance 

Manage  cost  by  reducing  head  count 

Contractor  examines  schedule  for  the 
program  to  additional  margin 
Amend  contract 

Projection  of  schedule  based  on 
hardware  and  l&T  progress 
Update  schedule  based  on  hardware 
and  l&T  progress 
APB  baseline  change 

Investigate  alternate  sites 

Accelerate  schedule  for  early  contract 
reward 

Accelerate  schedule  for  Early  contract 
reward 

Keep  Navy  and  OSD  informed  of 
progress 

Work  with  OSD  and  senior  Leadership 
Fund  DoD  Teleports 

Monitor  program  to  determine  when  to  TE 
launch  fund 

Work  with  Air  Force  Space  Command 
(AFSPC) 
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Comparative  Study  of  ISSUES 
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Issue:  Manual  Labels;  Topic:  Automated  Labels 


Topic  19 
Topic  17 
Topic  16 
Topic  15 
Topic  14 
Topic  13 
Topic  12 
Topic  10 
Topic  9 
I  Topic  8 
Topic  7 
Topic  6 
Topic  5 
Topic  4 
Topic  2 

♦ 


Issue  Cost  1 :  Repeated  inability  to  accurately  forecast  cost :  Topic  15 


Issue  Schedule  1:  Hardware  related  issue:  Topic  10, 12  19 
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AUTOMATED  ANALYSIS :  ACTIONS 
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ACTION:Manual  Labels;  Topic:  Automated  Labels 


Topic  14 
Topic  13 
Topic  12 

■  Topic  11 

■  Topic  10 

■  Topic  9 

■  Topic  8 

■  Topic  7 
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■  Topic  5 

■  Topic  4 

■  Topic  3 

■  Topic  2 

■  Topic  1 

■  Topic  0 
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IMAGE  ANALYSIS 


Conversion  of  program  Status 
matrix  to  numeric  format, 
Green  ->  1 
Yellow  ->  0 
Red  ->  -1 
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IMAGE  ANALYSIS  ALGORITHM 


Program  Status  (Chart  l> 
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1,1, 1,1, 1,1, 1,1, 1,1, 1,1, 

MDAP_A 

January 

2007 

uuuuuu 

U, i,i, i,i, U, i,i, U, 

1,1, 1,1, 1,1, 1,1, 1,1, 1,1, 

MDAP_A 

June 

2007 

uuuuuu 

0,0, 0,0, 0,0, 0,0, 0,0, 0,0,  0,0, 1,1, 1,1, 1,1, 1,1, 1,1, 

0,0, 0,0, 0,0, 1,1, 1,1, 1,1,  1,1, 1,1, 1,1, 1,1, 1,1, 1,1, 

0,0, 0,0, 0,0, 0,0, 0,0, 0,0, 

MDAP_A 

November 

2007  OAOAUUUU 

-1,  -1,-1,  -1,0, 0,0, 0,0, 0,0, '  1,1, 0,0, 1,1, 1,1, 1,1, 1,1, 

0,0, 0,1, 1,1, 1,1, 1,1, 1,1,  1,1, 1,1, 1,1, 1,1, 1,1, 1,1, 

0,0, 0,0, 1,1, 1,1, 1,1, 1,1, 

MDAP_A 

October 

2007 

1,  WMUUU 

0,-1, -1,-1,  -1,0, 0,0, 0,0,0,  '1,1, 1,0, 0,1, 1,1, 1,1, 1,1, 

0,0, 0,0, 1,1, 1,1, 1,1, 1,1,  1,1, 1,1, 1,1, 1,1, 1,1, 1,1, 

0,0, 0,0, 0,1, 1,1, 1,1, 1,1, 

MDAP_A 

September 

2007 

1,110,0,0,0,0,0,0,0,0,0, 

0,0,  -1,-1, -1,-1, -1,-1, -1,-1, 1,1, 1,1, 1,1, 1,1, 1,1,1, 

0,0, 0,0, 0,1, 1,1, 1,1, 1,1,  1,1, 1,1, 1,1, 1,1, 1,1, 1,1, 

0,0, 0,0, 0,0, 0,0, 0,0, 0,0, 

MDAP_A 

April 

2008  0AUUUUU 

0,0, 0,0, 0,0, 0,0, 0,0, 0,0,  0,0, 0,0, 0,0, 0,0, 0,0, 0,0, 

0,0, 0,0, 0,0, 0,0, 0,0, 0,0,  1,1, 1,1, 1,1, 1,1, 1,1, 1,1, 

1,1, 1,1, 1,1, 1,1, 1,1, 1,1, 

MDAPA 

August 

2003 

1,1,1, 1,1, 1,1, 1,1, 1,1,1, 

0,0, -1,-1, -1,-1, -1,1, 1,1, 10, 0,0, 0,0, 0,0, 0,0, 0,0,0, 

0,0, -1,-1, -1,-1, -1,1, 1,1, 11, 1,1, 1,1, 1,1, 1,1, 1,1,1, 

1,1, 1,1, 1,1, 1,1, 1,1, 1,1, 

MDAPA 

December 

2003 

1,1, 1,1, 1,1, 1,1, 1,1, 1,1, 

-1,-1, 1,-1,-1,-1,1,1,1,1,:0,  -1,-1, -1,-1, -1,1, 1,1, 1,1-1, -1,-1, -1,-1, -1,1, 1,1, 1,1, 1,1, 1,1, 1,1, 1,1, 1,1,1, 

1,1, 1,1, 1,1, 1,1, 1,1, 1,1, 

MDAPA 

July 

2003 

1,1,1, 1,1, 1,1, 1,1, 1,1,1, 

0,0, 0,-1, -1,-1, -1,-1, 1,1, 10, 0,0, 0,0, 0,0, 0,0, 0,0,0, 

0,0,0, -1,-1, -1,-1, -1,1, 1,11, 1,1, 1,1, 1,1, 1,1, 1,1,1, 

1,1, 1,1, 1,1,1, 1,1, 1,1,1, 

SAR  EXTRACTION:  COST  SUMMARY, 

FUNDING  SUMMARY,  TRACK  TO  BUDGET 


T<scal  Acquisition  Cosi  and  Quantity 


BY?QQ4*II 

8Y3aQ4$l4 

TY$M 

Apmoppiuiion 

SAR  Baseline 
Prod  Est 

C  ui  1 0ii  L  APB 
Production 
Qfi|BtWveThr«ftoi(i 

Current 

E&Liinale 

SAR  Baseline 
PretSEsl 

Cuireril  APB 
Production 
Objective 

Current 

Estimate 

RDT1E 

Procurement 

fts'awafy 
RfM-umrtg 
Mon  Rficurinq 
Support 
Qlher  Support 
Initial  Ep  ares 
MILCON 
Acq  Q&M 


3245  1  3245  2  3569  7 
2460.3  2460.3  2706.3 


36 36  2  4D40  Q 

3104.1  2831.3 
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PE  Extraction 

RDT&E  appropriations 


tYtfl  [Jin  Millions) 

All  Prior 
Y^arft 

FYJftl? 

Total  Program  Etemant 

4,174.406 

256.311 

072fl:  £HV=  SATMAf  Iwmtnafc 

500.077 

17.476 

073i:  mm 

15.20& 

4.165 

2472:AtolJbffM^^ 

3,573.210 

237.130 

FYM 


mm 


31.731 


FYW14  FYMM 


06.231 


21.077 


FYM14 

Total 


60.23' 


21.077 


FYJ015  FY2016 


33.116  24.324 


10.502  13.003 


hj  (MU05) 


FYJQ17  FYSfllS 


7.434  22160 


Cost  To 
Coniploto 


0.000  14.557 


Total 

[[Hi 


923  9.202  -1  9.202 

m 

9.45 

9  0000  C 

1.000  0.000  40.073 

COST  {5 
in 

Millions) 

All  Prior 
Years 

FY  2012 

FY 

EOlBi 

FY  2014 
Base 

FY  EC1M 
OCQ  q? 

ry  2014 

Tola! 

FY  2015 

FY  2016 

FY2D17 

FY  2016 

Cosl  To 
Total 

Complete 

Cost 

TolSl 

Ptrcjgram 

Element 

4,174  496 

m  8ii 

18848? 

66  231 

88231 

33188 

24  324 

7.434 

22 180 

CqiHiauIh 

9 

Cunlinuin^ 

072S 

EHF 

SATCQM 

Termnals 

536  077 

17  476 

J1  731 

21  077 

21  077 

16  502 

13693 

0 

14  557 

Conlinuin 

q 

Continuing 

oral 

FLTSMC 

OM 

15.239 

4.155 

10  526 

9.202 

9.202 

5.210 

1459 

0 

0.000 

0 

48.073 

247?' 

Mobile 

User 

Objective 

Sys 

(MU05) 

3.FJ3  ?1B 

?J7  ISO 

146 !??? 

36  SB? 

35  95? 

15  476 

7 16? 

7  434 

7623 

1130  91? 

4S153.S72 

Excel  Format 


UNC  Cl  IAKIXJTTE 

STAKE  YOUR  CLAIM 


-WWW.UNCG  JsDU  - 


LESSONS  LEARNED 


21 


•  Text  and  image  extraction/analysis: 

-  Existing  tools  and  technologies  can  support  large  scale  automated 
analysis  of  DAES,  PE  docs,  SARS 

-  Topic  models  built  from  MDAP  hub  data  seem  to  be  relevant  to 
neighbors. 

-  Challenges:  Formatting  and  Content  inconsistencies;  Missing  data 
reports. 

•  Funding  dependency  measure  can  be  obtained  from 

PE  and  SAR  data: 


Funding  amounts  for  the  MDAPs  as  captured  from  the  PE  documents 
and  respective  SARs  sometimes  don’t  match 

Funded  MDAPs  are  not  always  listed  in  the  Funding  Summary  page  of 
PE:  e.g.,  can  be  siphoned  through  a  non-MDAP. 
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FUTURE  WORK 


•  Large-scale  extraction  of  entire  set  of  available 
MDAP  data. 

•  Populate  Decision-theoretic  model 

-  State  features,  Action  Space,  Probability  Transitions 
and  Reward  Functions  (G,Y,R) 

•  Run  “what-if  simulations. 

•  Extend  to  Data  Network  and  MAIS  networks. 
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